Spontaneous symmetry breakings in two-dimensional kagome lattice 
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We study spontaneous symmetry breakings for fermions (spinless and spinful) on a two- 
dimensional kagome lattice with nearest-neighbor repulsive interactions in weak coupling limit, and 
focus in particular on topological Mott insulator instability. It is found that at ^-filling where there 
is a quadratic band crossing at F-point, in agreement with Ref.— , the instabilities are infinitesimal 
and topological phases are dynamically generated. At |-filling where there are two inequivalent 
Dirac points, the instabilities are finite, and no topological phase is favored at this filling without 
breaking the lattice translational symmetry. A ferromagnetic quantum anomalous Hall state with 
infinitesimal instability is further proposed at half-filling of the bottom fiat band. 



Introduction. — Recent interests have been revived in 
Fermi surface instabilities in connection with dynamic 
generation of topological Mott insulatorsir— . The first 
step is forwarded by Wu and Zhang^i^ where a new 
mechanism of generating spin-orbit coupling in strongly 
correlated, nonrelativistic systems is proposed. Atten- 
tions arc then focused in particular when the Fermi sur- 
face shrinks into discrete points, usually as band-crossing 
points (BCP), where the quasi-particle excitations arc 
not described by the Fermi liquid theory. Two examples 
are discussed in two-dimensional (2-D) honeycomb^ and 
checkerboard lattice^ with Cg and C4 rotational symme- 
try respectively. In the honeycomb lattice^, the BCPs 
have linearly vanishing density of states (DOS) at the 
Dirac points. Therefore only a full gap opening at the 
filling to the Dirac points could facilitate gaining energy, 
and the nodal phase becomes unstable only above finite 
critical interactions. While in the checkerboard lattice^, 
the BCP has quadratic dispersion with constant DOS in 
2-D, so that any gap opening at the BCP would help to 
gain energy and lead to the infinitesimal instabilities of 
the spontaneous breaking of rotational or time-reversal 
symmetries (TRS). 

Infinitesimal instabilities were proposed in 2-D systems 
with a quadratic BCP, which is protected by TRS and 
C4 or Ce rotational symmetryii^. In the explicit exam- 
ple with C4 symmetry in checkerboard latticoi, unequal 
next-nearest neighbor hoppings connected and not by di- 
agonal bonds are required to make the BCP quadratic, 
which makes it hard to search for real materials. In this 
work, we suggest to realize the infinitesimal topological 
instabilities in 2-D kagome lattice with Cg symmetry, 
where the kogomc compound herbertsmithite^ii appears 
to be a good candidate. The kagome lattice is a tri- 
angular lattice with three sites per unit cell, see FiglT] 
Within only the nearest-neighbor (nn) hopping, it has 
three bands, the top two bands cross at two inequivalent 
Dirac points and the bottom is a flat band. However, in- 
terestingly, by considering a small next-nearest neighbor 
(nnn) hopping, the flat band becomes dispersive, and its 
band structure shows both quadratic and Dirac BCPs at 



FIG. 1: Kagome lattice. There are three sites ("A, B, C") 
per unit cell, bi, b2 are bravais vectors. The arrows on the 
links represent currents respecting the reflection symmetry 
between up (dark) and down (gray) triangles, by breaking 
which together with TRS, a quantum anomalous Hall state is 
spontaneously generated. 



i- and |-fillings separately, which facilitates us to stress 
the differences of these two types of BCPs in a very same 
system. Although the magnetic properties of kagome 
lattice were studied extensivelySriS^ only little attention 
has been paid to its nonmagnetic insulating behavior, es- 
pecially the physics near the quadratic BCP. Recently, 
topological phases with broken TRS and quantized Hall 
conductance on kagome lattice have been studied for non- 
interacting fermionsii"— , while interacting fermions with 
broken symmetries are rarely explored. Finally, we pro- 
pose that a ferromagnetic quantum anomalous Hall phase 
with infinitesimal instability can be naturally realized at 
the half-filling of the bottom flat band. 

Symmetries and order parameters. — Before discussing 
specific models we first analyze the symmetries and de- 
fine order parameters. The symmetries of importance 
on kagome lattice under considerations are i) refiection 
symmetry between up and down triangles, ii) C3 rota- 
tional symmetry around the centers of triangles, and 
iii) time-reversal symmetry. We assume that the lat- 
tice translational symmetry wouldn't be broken, namely, 
wc do not consider the phases with finite q = Ki — K2 
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in the susceptibilities, where Ki and K2 are the two 
Dirac points. This is always true at i-filling because 
there is only one BCP at F-point, and the instabili- 
ties occur only at q = which preserves the lattice 
translational symmetry. Based on the symmetry con- 
siderations, we introduce the order parameters as below. 
Note that in the case of spinless fermions, since the or- 
der parameters, defined only in the sublattice space with 
= (Clj^ClgClfj) , have the same form as those in the 
spinful case, we only write down the order parameters in 
spinful case explicitly. For site order, the order parame- 
ters are N^in") = (^Ir" (g) where = 0, 1,2,3 

and = {CU^Cl^flctCUcUclci)- Here are 
Pauli matrices in spin space, and = diag(l 1 1), 
Rs = diag(l e^'^o e^*'^") are 3 by 3 matrices in sublat- 
tice space with 0^0 = ^- Among these order parameters, 
those of interests are n° for nematic state (spontaneous 
breaking of rotational symmetry) and n for nematic-spin- 
nematic (NSN) state^ii^. This is because the charge 
{N'^) and spin density wave (N) order parameters remain 
constants in the assumption of translational invariance, 
and in this assumption, the ferromagnetic states double- 
meant by N is competitive only at half-filling of the bot- 
tom flat band since we do not consider the spin-orbit 
coupling here. For bond order, the order parameters are 

Kid) = i^^lr" ® ir^'^l*) and S^^,^ = {^r'^ ® R:^"^]^), 
where the subscripts 'u' and 'd' indicate the order pa- 
rameters defined in up and down-triangles respectively, 
and 
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with = k • hi, i ~ 1,2. where bi and b2 are bravais 
vectors of kagome lattice (see FigH]) . The order param- 
eters under considerations are AJ^j.^-j which don't break 
the C3 rotational symmetry for bond order. 

Spinless fermions. — The model Hamiltonian for spin- 
less fermions with only nn repulsive interactions is 



Z^n.Uj (2) 



where t' - t > is a small nun hopping which is taken ac- 
cording to the transfer integrals in a natural spin- 1/2 
kagome compound known as herbertsmithite^ii. The 
nnn hopping term makes the bottom flat band disper- 
sive and quadratic at the F-point with which the Fermi 
surface crosses at ^-filling. However this term won't af- 
fect the low energy expansion near the Dirac points at 



|-filling. The nn repulsive interactions for side order 
will in general favor the nematic phase, which breaks the 
C3 rotational symmetry, with complex order parameter 
J'^o/„r.\ + e2''"o (nc) , where 



n ^ {TLA) + e"^"{jiB) + e^"^"{nc), wtiere (jix) 
X = A, B, C . There are two independent order parame- 
ters in this phase, the real part of n depicts the picture 
that the charge densities are equal on sites B and C but 
different on site A, while the imaginary part of n vi- 
sualizes that the charge densities on the three sites are 
all different. At ^-filling, the nematic order parameters 
split the quadratic touching at F-point with 27r Berry 
phas o^'^^ into two Dirac points with tt Berry phase each, 
thus open a gap at F-point to gain energy. At |-filling, 
the nematic order parameters pull the two Dirac points 
closer for small V . While at large V , the two Dirac points 
with opposite Berry phases annihilate each other and 
open a full gap to gain energy. Therefore the nematic 
order is always a competing phase at both fillings. The 
phase transition into nematic phase is of the first order. 
Our numerics show that the nematic phase will stabilize 
at large V for both fillings with Re(n) = ±1 respectively, 
where the charge densities reside all in A-site at -i-fiUing 
and equally in B- and C-sitcs at |-filling. 

For bond order, due to the frustration of the nn repul- 
sive interactions, we consider the possibilities of breaking 
reflection symmetry (or parity) between up and down tri- 
angles and TRS, while conserve the C3 rotational sym- 
metry ((5^(-£;-) = 0). The order parameters, ^u(d)} lead to 
four independent ones, Ai?± = i[Re(A„) ± Re(Ad)] and 
A/± = i[Im(A„) ± Im(Arf)]. Among these, Afl+ is only 
a renormalization of the nn hopping energy t and doesn't 
break any symmetry. While A/j_ breaks the parity and 
is responsible for the bond density wave (BDW) phase 
where the hopping amplitude differs in up and down tri- 
angles. This order parameter will remain the degeneracy 
of F-point at -i-fiUing, but opens a full gap at |-filling 
thus gain energy. For the imaginary parts which break 
the TRS generally, A/+ respects the parity and corre- 
sponds to a staggered flux picture, this order parameter 
won't open a gap at F-point, and only shifts up-and-down 
the energies at the two Dirac points. While A/_ breaks 
both parity and TRS giving rise to the quantum anoma- 
lous Hall (QAH) phase with topologically protected edge 
statesi^ and opens a full gap at both fillings. 

The mean-field Hamiltonian for spinless fermions is 



MF 



^0 + Ek *k [/^o (k) + /ii (k)] *k. The first term 



H 

in the Hamiltonian includes the nn and nnn hoppings, 
and 



hi (k) = 

2{nA) A: + Ade-^-= A„ -f A^e 

-V I A„ + A;e"^ 2(715) A*+Ade- 

A*+Ade-^-i A,„ + A*e"^ 2{nc) 



(3) 



withi;o = 2Ii[Rc(??,)2+Im(n)2]4-6F[A|j+ + A|_+A|^ 
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FIG. 2: (Color online). Finite temperature mean-field phase 
diagram for spinless fermions in kagome lattice at i-fiUing 
with t' It = 0.3. Thick and thin lines are respectively second 
and first order transitions. 



Aj_]. The mean-field free energy is obtained as 



f^(n,A„,Arf)=i?o-^^log(l 



(4) 



where /3 = l/fcsT, and is the number of unit cells. 
Since the order parameters arc classified by different sym- 
metries they usually order at different T^., therefore by 
minimizing the free energy with respect to the order pa- 
rameters separately, we could study the mean-field phase 
diagram at finite temperatures. The phase diagram at 
i-filling is shown in Fig. [21 where we see that the insta- 
bilities are infinitesimal at this filling, i.e., the symmetries 
arc broken at low temperatures for arbitrarily weak inter- 
actions. However this infinitesimal instability is absent in 
the phase diagram at | -filling due to the vanishing DOS 
at Fermi level. Our numerics show that at zero tempera- 
ture, there are two quantum phase transition points. For 
small V, the system remains in the semi-metal phase, till 
VcA ~ l.85t, the system goes into the BDW phase, and fi- 
nally stabilizes in the nematic phase for V > Vc2 = 2.55i. 
Note that the QAH phase never dominates in the phase 
diagram at |-filling near the Dirac points without break- 
ing the lattice translational symmetry. 

The infinitesimal instabilities are predicted in Ref.— 
in both checkerboard and kagome lattices with C4 and 
Ce symmetries, where the phase diagram is obtained in 
checkerboard lattice at half-filling. Here we see that for 
spinless fermions on kagome lattice, similar phase dia- 
gram is obtained at ^-filling by lifting the degeneracies 
of the flat band and make it quadratic at F-point. The 
infinitesimal instability at ^-filling on kagome lattice can 
be seen more clearly by projecting the mean-field Hamil- 
tonian into the two low energy states $t — (^t ^t-j j^g^j- 
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where Hq^ 



doao+dx 



-2{t + t')kxky, and d^ 



2t')P, dx = 
which have 



dzO'z with do 

d- wave symmetry, and Qi = (^Vz^), Q2 = (^Vaj^) are 
the effective nematic order parameters, A = (<I>Vy(i>) is 
the QAH order parameter. Here di's are PauH matrices in 
quasi-particle space. For weak coupling, the ground state 
of Hamiltonian ^ is QAH phase with the gap function 
at T = given by A cx exp[— ^^j^], where po is the 
constant DOS for a 2-D quadratic system. We see that 
at the phase boundary the interactions have infinitesimal 
instability, and the phase transition is of the second or- 
der. In contrast, by projecting the original Hamiltonian 
into the two high energy states near Dirac points, we 
have do = t + 2t' , dj: = ~\/3tkx, and d^ = VStky, which 
has p-wave symmetry. In this case, the gap function for 
QAH phase is A cx 1 — where A is the momentum cut- 
off, we see that at the phase boundary Vc — A, which is 
finite. We also perform the renormalization (RG) group 
analysis based on the projected interacting model at both 
fillings. Specifically, the result is very similar to that ob- 
tained in Ref.— at ^-filling. The interaction is marginal 
at the tree level, as expected from the constant DOS at 
the quadratic BCP; while it is marginally relevant at the 
one-loop level, which makes the system flow to strong 
coupling, supporting our mean-field results for symme- 
try broken phases. Except for the proportional constant 
which is mo del- dependent, the beta function is qualita- 
tively the same as that given in Refii. However at |- 
filling, the interaction is irrelevant at the one-loop level 
of the RG and higher loops are quite involved. 

Spinful fermions. — Now wc take the spin degrees of 
freedom into account and discuss the spin-^ fermions on 
kagome lattice. In addition to the nn repulsive interac- 
tions, we also include an on-site repulsive Hubbard term 
as well as a nn exchange term, 

{ij)a [ij\a 

+ n^fUii + niiij + J^Si ■ Sj (6) 



where n, = n^^ -I- n^j,. For site order, besides the ne- 
matic order as discussed in the spinless case where 
there is no spin order, there is a spin-triplet order pa- 
rameter n, nematic-spin-nematic phase, where there is 
no charge order. Different from the checkerboard lattice 
with C4 symmetry, the spin degrees of freedom won't re- 
store the Cg symmetry of kagome lattice in charge sector, 
therefore we expect both nematic and NSN phases in the 
phase diagram. With the spin degrees of freedom, the ne- 
matic order has contributions from both the spin-singlet 
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channel of Hubbard term ^nirii, and the nn repulsion 
term VniUj, which together form an effective nn repul- 
sion y = y — -J mimicing the nematic phase in the 
spinless model, therefore the nematic phase exists only 
in the region < U < AV in the U — V plane. For the 
NSN order, it comes only from the spin-triplet channel in 
the Hubbard term — -jSi-S^ where Sf = ^cJ^jT^^Ci/j. The 
NSN order parameters shift the two degenerate quadratic 
touchings into four Dirac points and open a gap at F- 
point thus gain energy. At |-filling, the NSN order pa- 
rameters split the two Dirac points into four at small U , 
one of the Kramer's pair will go closer and annihilate to 
open a gap as U increases, but the other two still remain 
linear touching and no full gap is opened, so that not like 
the nematic order, the NSN order is not competitive at 
|-fining. 

The bond orders all come from the nn repulsive interac- 
tions V. The spin-singlet ones A^^^^ are the order param- 
eters of QAH phase as discussed in the spinless model, 
while the spin-triplet ones Au{d) are the order parame- 
ters responsible for quantum spin Hall (QSH) phas o^'^^ . 
The QSH phase can be visualized as double QAH lay- 
ers with opposite flux, and in each layer, it is a QAH 
picture which breaks simultaneously the parity and the 
TRS. There are totally 12 independent spin-triplet order 
parameters which are A^_|_ = i [Re(AJj) ± Re(A^)] and 

= 5 ^^^^^e * = 1'2>3- Since 

the full Hamiltonian ([6]) has SU{2) symmetry we could 
choose A^^^j = (A°(.^),0,0, A^(^)) without loss of gener- 
ality, therefore the number of spin-triplet order param- 
eters is reduced into four. First, we note that Aj_ and 
A|j_ are degenerate with Aj_ and A^_ respectively, this 
is because at the mean-field level, the effective hopping 
satisfies = t^j-^-j and h\ = h\{u o d) which give the 
same spectrum due to the lattice symmetry. Therefore 
the QAH and QSH phases gain equal energies in the nn 
repulsive interactions V at both fillings, and both A^j^ 
are BDW order parameters. Second, the order parame- 
ter A|j_,_ breaks only the SU (2) symmetry and splits the 
energy bands with different spins by shifting the energy 
at F-point up-and-down a ma gnitude of V^|A|j_^| respec- 
tively, but keeps the quadratic point touched. So the gap 
opened at F-point of magnitude 2F|A|._^| is only due to 
spin-splitting without reducing the DOS. Finally, for the 
exchange term Hj ~ j[2ni — TiiUj — 2(c-Cj)(c^Cj)^], the 
UiTij term is only a shift of F to y — -j and gains equal 
energy for both QAH and QSH phases. However, the 
last term is the spin-singlct QAH order parameter, so 
that depending on the sign of the exchange coupling, the 
QAH phase will be favored if J > 0, otherwise the QSH 
phase will dominate if J < 0. 

The mean-field free energy at zero temperature is 
F = Eo + i^Y^^l^E, where = fy'|n°|2 + 

Sl^'P + 3^^E.=o,3 + A5i2_ + A^2 + 
Nf = 6L^f with the filling / = and |. By minimizing 
the free energy, the quantum mean-field phase diagrams 
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FIG. 3; (Color online). Zero temperature mean-field phase 
diagram for spin-i fermions in kagome lattice at (a) ^-filling 
(b) |-filling with t' /t = 0.3. Thick and thin lines are respec- 
tively second and first order transitions. The phase diagram 
at |-filling with small V and large U should not be taken too 
seriously. 

in the U — V plane are shown in FiglSJa) at i-filhng and 
in FiglSJb) at |-filling. At i-fiUing, it is seen again that 
all the broken symmetries have infinitesimal instabilities. 
We first notice that the spin-splitting phase (with the 
order parameter A|j_^ ) won't win over any other compet- 
ing phase in the entire U — V plane, which agrees with 
our analysis above. When < AV < U where the effec- 
tive interaction for nematic order V' is attractive, only 
the NSN and the QAH/QSH orders compete. Starting 
from the U = axis, the topological phases win first 
for arbitrarily weak nn repulsion and then be sup- 
pressed by the NSN phase at large U. When < U < AV 
where all nematic, NSN and topological phases compete, 
the NSN order is completely suppressed. The topologi- 
cal phase dominates at small V but then be taken over 
by the nematic phase, as seen in the spinless case, and 
the large on-site U helps to reduce the nematic order fur- 
ther. At |-filling, all the instabilities are finite again, and 
the topological phases arc not favored at all. Compared 
with the phase diagram in spinless case, we notice that 
at ?7 = 0, the multi-channel repulsions ftj ti and 
in the nn repulsion V in spinful model, facilitate the 
ordering of nematic phase relative to BDW phase, there- 
fore the BDW phase disappears in the U = axis, and 
the nematic phase begins to order at a finite interaction 
V > Vc = 1.25t. However, this multi-channel advantage 
is suppressed by large on-site Hubbard term, and the 
BDW appears for U > Uc = 2.44t. In the phase diagram 
at |-filling, we need to point out that the nodal phase at 
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small-y and large-C/ should not be taken too seriously. 
Since there are two Dirac points at this filling, the sys- 
tem could in principle gain energy by breaking the lattice 
translational symmetry and nesting between them, which 
is the case we do not consider in our discussions. 

Finally we discuss in brief the idea of ferromagnetic 
QAH state with infinitesimal instabilities at half-filling of 
the bottom band on kagomc lattice. If we consider only 
the nn hopping t, the bottom band is completely flat. 
In the presence of an on-site repulsive Hubbard term, its 
ground state is ferromagnetic at g-filling^. Now if we 
turn on the nn repulsion V, where the physics is effec- 
tively depicted by the spinless model at i-filling discussed 
before, the QAH state with infinitesimal instabilities will 
be favored in the ferromagnetic background. The ferro- 
magnetic QAH phase is a topological state with quan- 
tizcd Hall conductivity (t^^j, = it is stable even in the 
presence of a small nnn hopping t' . The ferromagnetic 
QAH state has been proposed in Hgi_j^Mnj,Te quantum 
wells with Mn-dopcd impuritiea^^, where a small mag- 
netic filed is still needed to polarize the Mn moments. 
However, on kagome lattice, we only need to tune the fill- 
ing to right to realize the ferromagnetic QAH state. The 



detail study of the ferromagnetic QAH state on kagome 
lattice will be present in a separate worki^. 

The material ZnCu3(OH)6Cl2, known as herbert- 
smithite, appears to be an excellent realization of the 
2-D spin-i kagome lattice^i^. This material consists of 
Cu kagome layers separated by nonmagnetic Zn layers, 
structurally with space group i?3m and lattice parame- 
ters c = 14.049i which is twice of a = 6 = 6.832i. The 
transfer integrals t and —t' of this material are 87meV 
and -lOmeV respectively^, which implies the opposite 
sign between nn and nnn hoppings and their ratio is 
quantitatively in agreement with that taken in our nu- 
merics. We suggest to realize all the symmetry broken 
phases on spin-i kagome lattice by tuning the filling, say 
replacing the Cl~ ions with sulfur (S) or oxygen (O), in 
this kind of material. 
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